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Abstract. Max-plus based methods have been recently explored for solution of first- 
order Hamilton- Jacobi-Bellman equations by several authors. In particular, McEneaney's 
curse-of-dimcnsionality free method applies to the equations where the Hamiltonian 
takes the form of a (pointwise) maximum of linear/quadratic forms. In previous works 
of McEneaney and Kluberg, the approximation error of the method was shown to be 
0{l/(NT))+0(yfr) where r is the time discretization step and ./V is the number of itera- 
tions. Here we use a recently established contraction result for the indefinite Riccati flow 
in Thompson's metric to show that under different technical assumptions, still covering 
an important class of problems, the error is only of order 0(e~ aNT ) + 0(t) for some 
a > 0. This also allows us to obtain improved estimates of the execution time and to 
tune the precision of the pruning procedure, which in practice is a critical element of 
the method. 

1. Introduction 

1.1. Max-plus methods in optimal control. Dynamic Programming (DP) is a gen- 
eral approach to the solution of optimal control problems. In the case of deterministic 
optimal control, this approach leads to solving a first-order, nonlinear partial differential 
equation, the Hamilton- Jacobi-Bellman equation(HJB PDE). Various methods have been 
proposed for solving the HJB PDE. We cite among all the finite difference schemes, the 
method of the vanishing viscosity of Crandall and Lions |CL84j . the discrete dynamic 
programming method or semi-Lagrangian method developed by Falcone [Fal87] and oth- 
ers |GD83j IFF941 ICFF04] , the high order ENO schemes introduced by Osher, Sethian 
and Shu |OS88|. IOS91] . the discontinuous Galerkin method by Hu and Shu |HS99j, the 
ordered upwind methods for convex static Hamilton- Jacobi equations by Sethian and 
Vladimirsky |SV03] which is an extension of the fast marching method for the Eikonal 
equations |Set99j . and the antidiffusive schemes for advection of Bokanowski and Zi- 
dani [B Z07] . These methods generally require the generation of a grid on the state space. 
This is known to suffer from the so called curse-of-dimensionality since the computational 
growth in the state-space dimension is exponential. 

Recently a new class of methods has been developed after the work of Fleming and McE- 
neaney |FM00j . see in particular the works of McEneaney [McE07j . of Akian, Gaubert 
and Lakhoua |AGL08] . of McEneaney, Deshpande and Gaubert [MDG08J, of Dower and 
McEneaney [DMllJ and of James et al. jSSMlOj . These methods are referred to as max- 
plus basis methods since they all rely on max-plus algebra. Their common idea is to 
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approximate the value function by a supremum of finitely many "basis functions" and 
to propagate the supremum forward in time by exploiting the max-plus linearity of the 
Lax-Oleinik semigroup. Recall that the Lax-Oleinik semigroup {S t )t^o associated to a 
Hamiltonian if (•, •) : W 1 x W 1 — > R is the evolution semigroup of the following HJB PDE 

(1) __^ + if ( X) V W ) = 0, (M)6l"x(fl,T], 
with initial condition 

(2) v(x,0) = <p(x), xeR n . 

Thus, St maps the initial function </>(•) to the function v(-,t). Among several max-plus 
basis methods which have been proposed, the curse-of-dimensionality-free method intro- 
duced by McEneaney |McE07j is of special interest. This method applies to the special 
class of HJB PDE where the Hamiltonian H is given or approximated as a pointwise 
maximum of computationally simpler Hamiltonians: 

(3) H(x, W) = max{H m (x, W)} 

with M. = {1, 2, • • • , M }. In particular, McEneaney studied the case where each Hamil- 
tonian H m is a linear /quadratic form, originating from a linear quadratic optimal control 
problem: 

H m {x,p) = {A m x)'p+^x'D m x + -p'Z m p, 
where (A m , D m , S m ) are matrices meeting certain conditions. Such Hamiltonian H cor- 



responds to a linear quadratic switching optimal control problem (Section 2.1) where the 
control switches between several linear quadratic systems. We are interested in finding the 
value function V of the corresponding infinite horizon switching optimal control problem. 



The method consists in two successive approximations (Section 2.4). First we approx- 
imate the infinite horizon problem by a finite horizon problem. Then we approximate 
the value function of the finite horizon switching optimal control problem by choosing an 
optimal strategy which does not switch on small intervals. 

We denote by (S t )t^o an d (S™)t^o for all m e M. respectively the semigroup corre- 
sponding to H and H m for all m e Ai. Let V° be a given initial function and T > be 
the finite horizon. The first approximation uses 5V[V ] to approximate V and introduces 
the finite-horizon truncation error: 

e (x,T,V°) := V(x) - S T [V°](x). 

Let t > be a small time step and N > such that T = Nr. Denote by 5V the semigroup 
of the optimal control problem where the control does not switch on the interval [0, r]. 
The second approximation approximates SVIV ] by {SVj^lV ]. The error at point x of 
this time discretization approximation is denoted by: 

e(x,r,N,V°) := S T [V°](x) - {S T } N [V°)(x) . 

The total error at a point x is then simply eo(x, T, V°) + e(x, r, N, V°). We shall see that 
— sup^jg^ S™. Therefore S T applied to a quadratic function corresponds to solving 
Riccati equations, requiring 0(|.M|72 3 ) arithmetic operations. The total number of 
computational cost is 0(\A4\ N n 3 ), with a cubic growth in the state dimension n. In this 
sense it is considered as a curse of dimensionality free method. However, we see that 
the computational cost is bounded by a number exponential in the number of iterations, 
which is referred to as the curse of complexity. In practice, a pruning procedure denoted 
by V T removing at each iteration a number of functions less useful than others is needed 
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in order to reduce the curse of complexity. We denote the error at point x of the time 
dicretization approximation incorporating the pruning procedure by: 

e^(x,r,N, V°) = S T [V°](x) - {V T o S T } N [V°](x). 

1.2. Main contributions. In this paper, we analyze the growth rate of eo(x, T, V°) as 
T tends to infinity and the growth rate of € Pt (x, r, N, V°) as r tends to 0, incorporating 
a pruning procedure V T of error 0(r r ) with r > 1. The error e(x, r, N, V°) in the absence 
of pruning is obtained when r = +oo. 

We show that under technical assumptions (Assumption 2.1 and 3.2), 

e (x,T,V°) = 0(e~ aT ), as T ->■ +oo 

uniformly for all x G W 1 and all initial quadratic functions V°(x) = \x ' Px where P is a 
matrix in a certain compact (Theorem 4.1 ). We also show that given a pruning procedure 
generating an error 0(r r ) with r > 1, 

€ Vt {x,t,N, V°) = 0(<r mln < 1 ' r - 1 >), as r -)• 

uniformly for all x G R n , N E N and ^° as above (Theorem 5.1). As a direct corollary, 
we have 

e(x,r,N,V°) = O(r), as r -»■ 
uniformly for all x G M n , N G N and l/ as above. 

1.3. Comparison with earlier estimates. McEneaney and Kluberg showed in |MK10[ 



Thm 7.1] that under Assumption 2.1, for a given V 



'0 



(4) e (x,T,V°) = O(^), asT-^+oo 

uniformly for all x G K n . They also showed |MK10l Thm 6.1] that if in addition to 
Assumption 2.1, the matrices S m are all identical for m G M, then for a given V°, 

(5) e(x,r,iV,y ) = O(V^), asr^O 

uniformly for all x G M n and iV 6 N. Their estimates imply that to get a sufficiently 
small approximation error e we can use a horizon T = 0(1/ e) and a discretization step 
r = 0(e 2 ). Thus asymptotically the computational cost is: 

0(\M\ 0{1/e3) n 3 ), ase^O. 

The same reasoning applied to our estimates shows a considerably smaller asymptotic 
growth rate of the computational cost (Corollary 7.1): 

0(\M\ 0{ - logie)/e) n 3 ), ase^O 

McEneaney and Kluberg |MK10j gave a technically difficult proof of the estimates Q 
and ([5]), assuming that all the S m 's are the same. They conjectured that the latter as- 
sumption can at least be released for a subclass of problems. This is supported by our 
results, showing that for the subclass of problems satisfying Assumption 3^, this assump- 
tion can be omitted. To this end, we use a totally different approach. Our main idea is to 
use the Thompson's metric to measure the error. It is well-known that the Thompson's 
metric defined on the space of positive-definite matrices is a Finsler metric and that the 
standard Riccati flow is strictly contracting in Thompson's metric (see |LW94| ILL07| ). 
However, we shall see that the Riccati equations appeared in the problem are indefinite 
and so we can not apply directly the contraction results. It has been shown recently 
in |GQ12] that the indefinite Riccati flows has a strict local contraction property in 
Thompson's metric under some technical assumptions. This local contraction result on 
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the indefinite Riccati flow constitutes an essential part of our proofs. We shall also need 
an extension of the Thompson's metric to the space of supremum of quadratic functions. 

Our approach derives a tighter estimate of e (x, T, V°) and e(x, r, N, V°) compared 
to previous results as well as an estimate of e Pr {x ) r, N, V°) incorporating the pruning 
procedure. This new result justifies the use of pruning procedure of error 0{r 2 ) without 
increasing the asymptotic total approximation error order. 

The paper is organized as follows. In Section [2j we recall the switching linear quadratic 
control problem and the max-plus approximation method. In Section |3j we recall the con- 
traction results on the indefinite Riccati flow as well as the extension of the Thompson's 
metric to the space of supremum of quadratic functions. In Sections [4] and |5j we present 
the estimates of the two approximation errors and part of the proofs. In Section |6j we 
show the proofs of some technical lemmas. Finally in Section [7j we give some remarks 
and some numerical illustrations of the theoretical estimates. 



We recall briefly the problem class and present some basic concepts and necessary 
assumptions. The reader can find more details in |McE07| . 

2.1. Problem class. Let M. = {1, • • • , M} be a finite index set. We are interested in 
finding the value function of the following switching optimal control problem: 



2. Problem statement 




where 



oo 



{/i : [0, oo) — > -M,/i measurable}, 




and £ is subject to: 



(6) 



^A^ + a^wt, £ = x. 



In the sequel we denote S m := ^-cr m (cr m )'. As in |McE07] . we make the following as- 
sumptions throughout the paper to guarantee the existence of V. 

Assumption 2.1. 



• There exists ca > such that: 



x'A m x sc -c A |x| 2 , Wx e R n , meM 



• There exists c a > such that: 



a 



.m 



^ C 



Vm e M 



• All D m are positive definite, symmetric, and there is cd such that: 
x'D m x < c D \x\ 2 , Vx eR n ,me M, 



and 




cdcI 
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2.2. Evolution semigroup. Define the semigroup: 

S'r[V r0 ](x) = sup sup J(x,T;V°;w, fi) 

where 

J(x,T;V°; w^) := J - ^\w t \ 2 dt + V {fr), 

T>t = {yU : [0, T) — > Ai,fi measurable}, 
and £ : [0, T] — > E is subject to ([6]). For each meM, define the semigroup {S™} t ^o- 

S™[V°]{x) := sup J(x,T;V ;™,)O, 
wew 

where 

^ m = m, te[0,T]. 

Note that for each t > 0, St is max-plus linear in the sense that for two functions 
Vi,V 2 : K n R, we have: 

(7) 5 t [F 1 + y 2 ](x) = ^[Vl](x) + S t [V r 2 ](x), \/xeR n 

(8) ^[sup{T4,F 2 }](x) = sup{5 t [y 1 ](x),^[\/ 2 ](a;)}, Va; £ M n 

2.3. Steady HJB equation. For any S £ (0,7), define 

(9) G 5 := {V semiconvex , War) < 7 - |x| 2 , Vx}. 

Then the value function is the unique viscosity solution of the following corresponding 
HJB PDE in the class G s for sufficiently small S |McE07| : 

(10) = -H(x,W) = -max# m (x,W). 
where 

H m (x,p) = (A m x)'p+-x'D m x + ^p'TTp. 
It was shown in [McE07j that for 5 sufficiently small and V° £ Gs, 

(11) lim S T [V°] = V 

T— >oo 

uniformly on compact sets. 

2.4. Max-plus based approximation. We review the basic steps of the algorithm 
proposed in |McE07j to approximate the value function V. Firstly using (11) we are 
allowed to approximate V by <St[V°] for some sufficiently large T. We then choose a 
time-discretization step r > and a number of iterations N such that T = Nt to 
approximate St by 

S T = {S T } N ~ {S T } N 

where S T = sup S™. At the end of N iterations, we get our approximated value function 
represented by: 

y~ sup S l T N ■■■Si}[V }. 

ill— ,*jv 

If we choose V ( •Xj ^ 2 35 Px as a quadratic function, then the approximated value function 
will be the maximum of |A"f| quadratic functions. This so-called curse-of-complexity 
can be reduced by performing a pruning process at each iteration of the algorithm to 
remove some quadratic functions, see [MDG08J. 
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2.5. Approximation errors. As pointed out in [MK10J, the approximation error comes 
from two parts. The first error source is 

e (x,T,V°) := V(x) - S T [V°](x). 

This is due to the approximation of the infinite-horizon problem by a finite-horizon prob- 
lem. The second source of error is 

e(x,r,N,V°) := S Nr [V°](x) - {S T } N [V°}(x), 



due to the approximation of the semigroup by a time-discretization. In Section 2.4 we 
mentioned that in practice a pruning procedure is needed so as to reduce the number of 
quadratic functions. More precisely, let {fi}i^i be a finite set of quadratic functions and 

(12) /=SU P /;. 

16/ 

A pruning operation V applied to / produces an approximation of / by selecting a subset 
JC/: 

sup / 3 . 

If we take into account the pruning procedure, then the second error source should be 
written as: 

(13) e^(x,r,N,V°) = S Nt [V°](x) - {V T o S T } N [V°](x), 

where V T represents a given pruning rule. We mark the subscript r since it is expected 
that the pruning procedure be adapted with the time step r. In particular, we say that 
V T is a pruning procedure generating an error 0(r r ) if there is L > such that for all 
function / of the form (12), 

(14) PAf] < / < (1 + Lr r )P T [f]. 

The special case without pruning procedure can be recovered by considering r = +oo. 

3. Contraction properties of the indefinite Riccati flow 

Before showing the main results, we present here an essential ingredient of our proof: 
the contraction properties of the indefinite Riccati flow. 

3.1. Loewner order and Thompson's part metric. We recall some basic notions 
and terminology. We refer the readers to |Nus88j for more background. 

We consider the space of n-dimensional symmetric matrices S n equipped with the 
operator norm || • ||. The space of positive semi-definite (resp. positive definite) matrices 

is denoted by S„ (resp. S n ). The Loewner order on S n is defined by: 

A^B^B-AeS+. 
For A ^ B we define the order intervals: 

[A, B] := {P G S n \A < P < B}. 

For Pi, P 2 G S n , following |Nus88] . we define 

M(P 1 /P 2 ) := mi{t > : P 1 < tP 2 } 

Definition 3.1. The Thompson part metric between two elements Pi and P 2 of is 

d T (P u P 2 ) := log(max{M(Pi/P 2 ),M(P 2 /Pi)}). 



3.2. Contraction rate of the indefinite Riccati flow. For each m G Ai, define the 
function $ m : S n — » S n : 

(15) $ m (P) = (A m )'P + PA m + P£ m P + D m . 
Associated to each $ m we define the flow map by: 

M™[P ] = P(t), te[0,T) 
where P(t) : [0, T) — > S n is a maximal solution of the following initial value problem on 

(16) P = $ m (P), P(0) = P . 

When V°(x) = \x' P^x with P G S n , a classical result [YZ99J states that 

(17) Sr[V°](x) = ^x'M™[P }x, te[0,T). 



The standard Riccati equation refers to a vector field of the form (15) with — E m and D m 
positive semi-definite. Here we are concerned with the indefinite Riccati equation since 
the matrix coefficient S m is positive semi-definite. 

The contraction property and the contraction rate calculus of the standard Riccati flow 
in Thompson's metric have been given in [LW94J and |LL07j . However, their approach 
depends on the algebraic property of the associated symplectic operator, which fails 
in the indefinite case. In |GQ12|, the authors give a general explicit formula for the 
local contraction rate of a flow, in Thompson's metric, from which it follows that under 
additional constraints on the matrix coefficients, the Riccati flow is still a local contraction 
in the indefinite case. Below is the additional assumption needed to apply this new 
contraction result: 

Assumption 3.2. There is mp > such that 

x'D rn x > m D \x\ 2 , Vx G W n , meM 

and 

c 2 I 

^m D > (c A - yc 2 A - c D c2/ 7 2 ) 2 . 

In the sequel we denote 

x 7 2 (ca ~ 74 - c D clh 2 ) , — 

Ai = — , A 2 := yJm D Y/ci- 



c 2 



Remark 3.3. Under Assumption |3.2[ we can choose e > sufficiently small so that 

(18) M™[0] ^ el, for some t > 0, m G M. 

Since $ m (0) = D m ^ mpl for all m G Ai, we can let e be sufficiently small such that 
$ TO (e/) ^ for all m G Ai. Besides, for any A G [Ai,A 2 ), we have $ m (A7) ^ for all 
m G Ai. Then it follows from a standard result on the Riccati equation that: 

(19) M t m [P ] G [el, XI], Vm G Ai, t ^ 0, P G [el, XI}. 
The main ingredient to make our proofs is the following theorem: 



Theorem 3.4 (Corollary 4.6 in |GQ12| ). Under Assumptions 2.1 and 3.2, for any X G 
[Ai, A 2 ) ; there is a > such that for all Pi, P 2 G (0, Ai], 

rf T (M™[Pi],M t m [P 2 ]) < e- at d T (P 1 ,P 2 ), Vt ^ 0,m G Ai. 
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3.3. Extension of the contraction result to the space of functions. Now we extend 
the definition of Thompson's metric to the space of non-negative functions. For two 
functions /, g : IR n — > R, we consider the standard partial order " ^" by: 

f^g^f(x)^g(x), Vxel", 

which coincides with the Loewner order on the set of quadratic forms. Similarly, for 
/, g : R n -> R+ we define 

M(f/g) := inf{t >0:f^tg} 

We say that / and g are comparable if M(f/g) and M(g/f) are finite. In that case, we 
can define the "Thompson metric" between f,g : R n — >■ R+ by: 

(20) d T (f,g) = log{msx{M(f/g),M{g/f)}). 
Then the following lemma can be easily proved using the definition: 

Lemma 3.5. Let /, g : M n — >■ R + be given by pointwise maxima of non-negative functions 

f ■= sap fi, g := sup#i 
iel iel 

Then 

(21) d T (f,g) ^ sup d T (fi,gi). 

iel 

The following result is a consequence of the order-preserving character of the Riccati 



flow and of the contraction property in Theorem 3.4 



Lemma 3.6. Under Assumptions 2.1 and 3.2, let A G [Ai, A2) and e > such that (19) 
holds. Then there is a > such that for any two functions V\ and V% of the form: 

Vi(x) = sup -x'PjX, V 2 (x) = -x'Qx, 
jeJ 2 2 



where Q, Pj G [el, XI] for all j G J, we have 



X 



d T {SlJ N ■ ■ ■ S^ N [V X \, Slfa ■ ■ ■ S? /N [V 2 }) < e"* log(-) 

for allt^O, N G N and {i u ■ ■ ■ , i N ) G M N . 

Proof. For all P,Q<E [el, XI], by ( [i~9| ) and Theorem 3.4 we have 

d T (M}f N ■ ■ ■ Ml) N [P},Mlf N ■ ■ ■ M*j N [Q] ) < e- at d T (P, Q) 
for all t ^ 0, N G N and (ii, ■ ■ ■ ,in) G Ai N . Now by the max-plus linearity of the 



< sup d T {Mlf N ■ ■ ■ Ml) N [P 3 ],Mlf N ■ ■ ■ Ml) N [Q\) 

< e~ at sup d T (Pj, Q) ^ e~ at log( 

jeJ 



semigroup (|8j), Lemma 3.5 and the relationship between the semigroup and the flow (17), 
we get 

J 

N 

□ 



4. Finite horizon error estimate 

We first study the finite horizon truncation error €q(x, T, V°). Below is one of our main 
results: 



Theorem 4.1. Under Assumptions 2.1 and 3.2, let X G [Ai,A2) and e > such that (18) 
and (19) hold. There exist a > and K > swc/i i/iai, 

e {x,T,V°) ^ Ke~ aT \x\ 2 , Vx, 

/or allT>0 and V°(x) = \x' 'P x with P G [el, XI}. 

The remaining part of the section is devoted to the proof of the above theorem. We 



shall need the following technical lemma. The proof is deferred to Section 6.1 



Lemma 4.2 (Approximation by piecewise constant controls). Let V° : lR n — > 1R be a 
given locally Lipschitz function. For any T > we have 

S T [V°}=sup sup S^-^S^V }. 



N ii,— ijv 



From now on we make Assumptions 2.1 and 3.2 We also fix A G [Ai,A2) and e > 
satisfying (18) and (19). 

Remark 4.3. Since the interval [el, XI] is invariant by any operator {S^^q^Mi h is 
direct from Lemma [4.21 that 

A, 



(22) 



M < S T [V"}(x) ^ -\x\\ VT > 



for all V°(x) = \x'Px with P G [el, XI}. 

Corollary 4.4. The value function V is a pointwise supremum of quadratic functions 



V(x) = sup -x'PjX 
jeJ 2 



where Pj G [el, XI} for all j G J. 
Proof. By definition, we have: 



V(x) = sup iSy[0](a;), Wx. 

T>0 



By (18), there is to > and m G Ai such that 

M£[0] ^ el. 
Besides, by the monotonicity of the semigroup, 

S T [S%[0}}(x)^S T [S to [0}Kx), Vx,T>0 

and 
Since 

we get that: 



S T [0}(x)^S T [S^[0}](x), Vx,T>0. 



V(x) = supS r [0](x) = sup^ T [^ io [0]](x), Vx, 

T>0 T 



mn>Sr[S2[0]](x) ^ V(x) ^ sup S T [S? o [0}](x), x. 

T T 



Hence by Lemma 4.2 



V(x) =Bupfl!r[5^[0]] =su P su P sup S? /N ■ ■ ■ 5£ /JV 5£[0]. 
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Now using the invariance of the interval [el , XI] in (19), we know that 

M*/ /N ---M£ /N M£[Q]e[eI,XI], 

for all T > 0, N G N and i\, . . . ,in G M.- Consequently V is a pointwise maximum of 
quadratic functions \x'PjX with Pj G [eJ, XI]. 

Using the above lemma we show that: 

Proposition 4.5. There is a > such that for all V°(x) = ^x'P^x with P G [el, XI], 



□ 



d T (V,S T [V ]) ^ e- aT \og(-), VT > 0. 



A, 



Proof. By Corollary |4.4[ the value function V is a pointwise supremum of quadratic 
functions: 

V(x) = sup -x'PjX 
jej 2 

where Pj G [el, Ai] for all j G J. Let any V°(x) = \x'P a x with P G [e/,AJ]. By 
Corollary |3.6[ we have: 



ms%n ■ ■ ■ ssyn s^v • • • 4v^°d < e-^dog 



A, 



for all T ^ 0, iV G N and (i\, ■ ■ ■ , zjy) G At . We also know from Lemma 4.2 that 



and that 



V = S T [V] = sup sup S* N . . . S^ /N [V], 

N ii,...,ijv 



S T [V°] = su P sup S*" . . . S% /N [V°] 

N ii,...,ipf 



Therefore by Lemma [3. 5[ 

d T (V,S T [V ]) = d T (S T [V],S T [V ]) 

^ sup sup d T (s^ /N ■ ■ ■ s^ /N [v], sr T % ■ ■ ■ S^ /N [V ]) 

^e- aT log(^). 



Now we have all the necessary elements to prove Theorem |4.1| 



□ 



Proof of Theorem 4J_' Let any V°(x) = ^x'PqX with Pq G [el, XI]. By Proposition 
and (20), there is a > such that 

V(x) < e e ~ aTloe{x/e) S T [V°](x), VT > 0,x G M n . 
Thus there is constant L > such that 

V{x) < (1 + Le- QT )5 T [U°](x), VT > 0,x G M n 

This leads to 

e (x,T,y°) ^ Te- QT S T [U ](x) < ^e" aT |x| 2 , VT > 0, x G M n . 



4.5 



independent of P G [eJ, AJ]. 



where the last inequality follows from (|22|). It is clear that the constant K — ^ is 

□ 



10 



5. Discrete-time approximation error estimate 



In this section we analyze the discrete-time approximation error € Pt (x, t, N, V°). Our 
main result is: 

Theorem 5.1. Let r > 1. Suppose that for each r > the pruning operationV T generates 
an error 0(t t ) (see {14:)). Under Assumptions 2.1 and 3.2, let A G [Ai, A2) and e > 
such that (19) holds. Then there exist tq > and L > such that 

e v ^(x,T,N,V°) ^ Lr min{1 > r - 1} \x\ 2 , Mx, 

/or all N eN, t ^ r and = §x'P :r mtfi P e [el, A/]. 



The remaining part of the section is devoted to the proof of Theorem 5.1 We first 



state a technical lemma which is proved in Section 6.2 



Lemma 5.2. Let /C C S n be a compact convex subset. There exist tq > and L > such 
that 

S T [V°]{x) ^ S T [V°}{x) + Lt 2 \x\ 2 , Vx, 
for all r G [0, r ] and V°(x) = \x' ' PqX with P G /C. 

Now we take into account the pruning procedure and analyze the error of the following 
approximation 

Below is a direct consequence of Lemma 5.2 and (19). 



Corollary 5.3. Let e, X, r and V T be as in Theorem 5.1 Then there exist tq > and 
L > such that: 

Sr[V°]{x) *C (1 + Lr min ^ 2 ^)P r o 5 T [y°](ar), Vx, 
/or a// t G [0, r ] and V°( 'P x with P G [e/, XI] . 

We are ready to give a proof of Theorem |5.1| 



Proof of Theorem 5.1. Denote s = min{2,r}. Let any A' > such that 

A < A' < A 2 . 

Denote 5 = X'/X. Consider the two compact convex subsets /Co = [el, XI] and /Ci 
[el, X'l}. It is easily verified that: 

$ m (A7) < 0, Vm G M. 

Therefore for all P G JC , Pi G /Ci, t ^ and m e M, 

(23) Mr[Po]G/C , iWf^lG/d. 



5.3 



there is tq and L > such that for all r G [0, To] and V = ^x'Px with 



By Corollary 
PG/C X : 

(24) 5 T [y°] < (1 + £t s )P t o 5 T [V°] . 

Let To > be sufficiently small such that: 

(1 + Lt s )^=^ < 5, Vtg[0,t ]. 

Let any V A °(x) = |x'Pox with P G /Co and t G [0, To], we are going to prove by induction 
oniVeN the following inequalities: 

rOl ^ n , r^s\l+e- QT +---+e-( Ar - 1 ) aT 
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< (1 + Lt s 



{P r oS/[7°], VA^gN. 



The case N = 1 is already given in (24). Suppose that the above inequality is true for 
some k G N, that is, 

S kT [V°]^L k {V T osnpS^} k [V°] 

m 

where L k = (1 + Lr s ) 1+e aT+ - +e {k 1)aT . We denote by I k C M k the subset such that 

{v T o sup s™} k [v°] = sup s l T h ■ ■ ■ [V°] . 

Thus, 

(25) S kT [V ]^ sup L k S? ■ ■ ■ S?[V }. 



From (23), we know that for all (h, ■ ■ ■ ,i k ) £ I k 

(26) M^-.-M^fPo] e/Co. 
Besides, 

1 < L fc < (1 + Lr s )^^ < 5. 

Thus for all (ix, ■ ■ ■ , i k ) G I k , 

(27) L A (M^---M^[P ]) e/d 
Recall that 

L k Sl k ■ ■ ■ Si}[V°](x) = yx'(M^ ■ ■ ■ M?[P ])x, 

then by applying ( J25) ) and (24), we obtain that 

S (k+1)T [V } = S T [S kT [V ]} 

< sup S T [L k S? ■ ■ ■ S?[V }} 

{26) (ii,- ,i fc )G/fc 

*C sup (1 + Lr*)P T o ^.[[LfcSj.* • • ■ S; 1 [V ]]] 



Now by Theorem 3.4, there is a > such that for all P\,Pi G /Ci and m e M. 

d T (M™[P x ],M?[P 2 ]) < e- aT d T {P u P 2 ) 



Therefore from (26) and (27) we get that for any (ix, . . . ,i k ) E I k and m e M 

d T (M™M? ■ ■ ■ M* 1 [P ], M™[L k (Mjf • • • m; 1 [P ])]) 

< e- aT d T (M? ■ ..M*[P ],L k (M? ■ ■ • M*[P ])) = e~ aT \ogL k . 

This implies that 

M™[L k (Mi k ■ ■ • M* 1 [P ])] < Ll aT M™Mi k ■ ■ • M* 1 [P ], Vm G .M 

which is, 

S?[L k S!* ■ ■ • S; 1 [F ]] < aT S™Si k ---Si 1 [V°], Vm G M. 
Therefore we deduce from the inequality ( 28 ) : 

S( k +i)r[V } < (1 + Lr s )P r [ sup 5-[L fe ^ • • • [V ]]} 

meM,{ii,— ,ifc)ei* 

< (1 + Lr s )Lf " T P T [ sup • • • S; 1 [V ]] 

meM,(ii,— ,ife)e4 
= (1 + Lr s ) 1+e " QT +-+ e - fc ^{p r o S r } fe+1 [K ]. 

Thereby we proved that 

5jv r [V°] < (1 + Lt^^^P o Sr^fV ], ViV G N. 
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Note that 



lim 

T^0 + 



'1 + Lr s )i--^ - 1 



-8-1 



L 

a 



from which we deduce the existence of tq and K > such that for all r G [0, To], N G N 
and V°(x) = \x'Px with P G [el, XI] 



{s T } N [v } < (l + KT^iPo^fty ]. 



i N r T/ 0i 



which leads to 



If A 



e pT (x,T,iV,r ) < X T ™{l.--l}^o5 T f[y ] < li^ r mm{l,r-l}| x | 2> 



□ 



Remark 5.4. It should be pointed out that the crucial point is having a > 0. If this is 
not the case (a = 0), then the iteration (28) only leads to: 

d T (S NT [V°], {V T o S T } N [V ]) < LNt s , ViV G N. 
6. Proofs of the technical lemmas 



6.1. Proof of Lemma 4.2, For two functions v G XV we consider the metric z/) 
defined by the measure of subset on which the two controls \i and v differ from each 
other: 

cT 



(29) 



d(fi, v) 



dt. 



The proof of Lemma |4.2| needs the next lemma. It shows that the objective function is 
continued on the variable fi G XV with respect to the metric d defined in (29). 

Lemma 6.1. Let V° : W 1 — > R be a locally Lipschitz function. Let x G M n and T > 0. 

Given \i G XV and ty G Wt, /or any e > 0, i/iere is 5 > suc/i i/iai 

| J(z, T; \/ ; /I, w) - J{x, T; V°; ft, w) | < e, 
/or a// /i G XV stzc/i £/iat /2) ^ 5 . 
Proof. Let any p, G XV and denote: 

5 = d(n, p). 

Let £ and £ be respectively the solutions to (Jsl) under the control (ji, w) and (fx, w). Thus 



6-6 



Denote 



L = max(max ||^4 m ||, max |<7 m |,max \\D r 



(\{ a \ + \w 8 \) 2 ds) x ' 2 ) 



We have: 



|& - 61 < £ l AMs 6 - ^ S 6I + l^ s 6 - ^ As 6l + 1^ - o-**\\w s \ds 

< / *L|6-6|ds + £ WP" S - ^ II 16 1 + \a^ -a^\\w s \)ds 

< Jo L ^ " ^l rfs + 2L Jo WI6I + \w,\)ds 

< J!; m - i\ds + 2L(/ i \^dsfi\si(u + \w s \n 12 

^ f*L\£ s -i s \ds + 2L 2 62, VtG[0,T]. 
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By Gronwall's Lemma, 

|& -it\ < 2L 2 dh Lt ^ Vte[0,T\. 

Then 

Ht\ ^ sup <L,VtG [0,T]. 

ie[o,T] 

Note that L is independent of /2. Now by the local Lipschitz property of V° and the 
boundedness of £ and £, there is L > such that: 

|y°(£r) - 7 (|r)| < L\£ T - | r | < L6$ 

Besides, 

< Jo" - it)\ + |gZ>"*(& - 6)1 + |g(U^ - D*%\dt 

< L((Js +5) 

Thus there is a constant L independent of /2 such that: 

\J(x,T; V°;lx,w) - J(x,T;V°;fi,w)\ < L(c^ + 5) 
whence for any e > there is Sq > such that 

\J(x,T; V°;li,w) - J(x,T;V°; fx,w)\ ^ e 
for all fx G V T such that d(/x,fx) ^ <5 - □ 



Using this, we can prove Lemma 4.2 



Proof of Lemma \4 ■ %\ Let V° be a locally Lipschitz function. Fix x G R™. Let /x G D T 
and io G W r be ^-optimal for SV[V ](a;), that is: 

(30) S T [V°](x) < J(x,r;V ; / i, W ) + ^. 



By Lemma 6.1, there is 5o > such that: 

(31) \J(x,r;V°;j2,w)-J(x,T;V ; fj,,w) \ ^ | 

for all ft G D T such that d(fi,fi) ^ 5q. Now it remains to prove that there is at least 
one piecewise constant function fx G T> T such that d(fx,fl) ^ 5q. To this end, by Lusin's 
theorem [Fol99j, there is a compact K C [0, r] such that 

1a: > t - 5 

and the restriction of it on K is continuous, thus uniformly continuous. Let 5 > such 
that for all t,s & K and |t — s| ^ 5, 

\Kt)-Ks)\ < \ 

which implies 

H(t) = /x(s). 

Now let Nq G N such that < 6. We construct a piecewise constant function fx G X? T 
as following. For i G {0, 1 • • • , Nq — 1}, let 

r,(—^\ - S if there iss6i{n t4 r ' W r ) 

M iV j ~ \ 1, else 

14 



and 

Since /i(s) = fi(t) for all s,t G K D [^r, ^r), it follows that 



Thus 



H(t) = /2(t), Vt G K 



So ^ <5 and /i is constant on interval [j^t, j^ T ) for all i G {0, 1, • • • ,N — 1}. 

Hence, by (31 ), 



J(z,t;VV,«0< J(x, r; /i, w) + ^ sup ■ ■ ■ S? /No \V°](x) + \ 



Now by (30), we get 

S T [V°] (x) ^ sup S% ■ ■ ■ SI) Nq [V°] (x) + e < sup sup ^ • • ■ S? /N [V°] (x) + e. 
This is true for any e > 0, we conclude that: 



S r [V°] (x) = sup sup 5^ • • • S l r ) N [V°] (x) 



for all xeR n . Thus 



S r [V°] = sup sup K% ■ ■ ■ S^V }. 



N ii,— ,tN 



□ 



6.2. Proof of Lemma 5.2 The proof of Lemma 5^ shall need the following estimates: 

Lemma 6.2. Le£ /C C S n be a compact convex subset. There exist r > and L > s«c/j 
that 

\\M?[P] -P- T$ m {P )\\ < Lt 2 + Lr\\P - P || 
for all P, P Q G K, t G [0, r ] and mGM. 

Proof. Let r > such that for all P G /C, m G .M, the Riccati equation 

P = $ m (P), P(0) = P, 

has a solution in [0, r ]. Therefore, 

K := {M t m [P] : t G [0,r ],P G /C,m G X} 

is compact. Besides, for P G /C and m G .M, the function M m [P] : [0,r ] — >■ S n is twice 
differentiate in the variable t and it satisfies: 

M?[P] = <S> m (M™[P}), M™ [P] = D$ m (M™[P]) o $ m (M t m [P]), i G [0,t„]. 

By the mean value theorem, for all P, Po G /C and r G [0, To] 

||M™[P]-P-r$ m (P)|K sup \\D$ m {M™[P])o<S> m {M™[P])\\T 2 

te(o,T) 



and 



Let 



|$ m (P)-$ m (P )|| < sup||P$ T 

Qgk: 



|P-Pn 



L = max{sup sup ||P$ m (P) o $ m (P)||,sup sup ||P$ m (P)||}, 

m pg£ m P£lC 
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then we have 

\\M?[P] -P- r$ m (P )|| < Lr 2 + Lr\\P - P | 
for all P, P G JC, r G [0, r ] and me M. 



□ 



Using Lemma [6.2| we give a proof of Lemma 5.2 



Proof of Lemma 5. 2 , Let any < 5 < 1 and /C C S n be the compact convex defined by: 



By Lemma 

(32) 

Let 



6.2 



JC := conv(Up oe /c-B(P ,<$))- 
there exists t 1; Li > such that for all m G Ai, P, P G AC and r G [0, t x ] 
M T m [P] < P + T$ m (P ) + [L x r 2 + L^HP - Po||)/. 



L 2 

TO = 



sup{||$ m (P)|| :meM,PeJC}, 
max(Li, L 1 L 2 ), 

5 



mm 



■ 2L 2 ' Y 2eL ' Li ■ 

Let any iVeN ,(ii, G .M N , r G [0, t ] and V°(x) 

going to prove by induction on k G {1, • • • , iV} that: 



\x'PqX with P G /C. We are 



(33) 



1V1 t/N 



M 



T 



T 



T/N [Po] <Po + jj*iAPo) + ■■■ + + A>(i + ^ 



1 



AT2 



When k = 1, since G [0, t ] and P G /C, by (32) we get: 

M^Po) ^P + ^ n (Po) + L 1 (^) 2 / 

^Po + ^ 1 (Po) + L (l + ^)(^) 2 /. 
Suppose that (33) is true for some k G {1, • • • ,N — 1}. That is: 
(34) Mt k 

where A k = ^ h (P ) + ■■■+ %$i k (Po) + L (l 



1 ^ r2fc2 J. Since 



A' 



N 2 



\\A k \\ <: %L 2 + L (1 1 ^ kr2k2 



< tL 2 + L er 2 < 5, 



AT2 



we have that Pn 



/TV 



A fc G /C and by (|32j): 
A fc ] ^P 

< P + ^(Po) + • • • + %*i k {P ) + 



A fc + ^$ ife+1 (P ) + (LijfS + L^\\A k \\)I 



1 \kr 2 k 2 



N ' N 2 

+^ ik+1 (P ) + L^L + L^[fL 2 + L (l + 



P + ^$ n (P ) + ■ ■ ■ + ^(Po) + 

^P + ^$ n (P ) 



^[L (l + ^) fc fc 2 + L x + L x L 2 k + LxLoCl + £) fc2 f K 



£<& <fc (p ) + £$ <im (P ) 



[^o(l + ^) fc (A; 2 + A ; + l) + Lo(l + ^|]/ 
< P + fc* ix (Po) + ■ ■ ■ + + i^+^o) 



AT 

T 2 (fc+l) s 
AT2 



^o(l + 



Thus, by (34) and the monotonicity of the flow: 
K7nK)n ■ ■ ■ K)n\Po] < [P + Aj 



r/N L- 1 UJ ^ J "-V//V L J U -r ^fcj 

< P + + ■■■ + ff*i>(Po) + W^(Po) + L o(l 

We conclude that: 



1 \k+\ r 2 (k+lf 
N ) N 2 



L. 



M iN 

IV1 t/N 



IV1 t/N 



[P ] ^ P + ^$ n (P ) + ■ ■ ■ + %$ iN (Po) + ^L T 2 I 
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Denote: 



g(x) = sup )-(x'Pqx + x'$ m (P )x). 



By Lemma < oJl_ we have that 

P + r$ m (P ) < M T m [P ] + £ir 2 /, Vr G [0, r ],m G M.. 

That is 



<7(x) < S™[V°}(x) + ^r 2 \x\\ Vr G [0, Tq], 771 G M 



Therefore, 

Q«JV 

D r/JV ' ' 



< Kx'Pqs + ^^^(Po)^ + ■ ■ • + ^x'$ ijv (P )a; + eL r 2 \x\ 2 ) 
^ g(x) + eJ fr 2 \x\ 2 

^ sup m S?\V°](x) + Lt 2 \x\ 2 , ViGl" 

where L = eL °+ Ll is clearly independent of V^ , N, - • • ,in) and r ^ r . We conclude 
that: 



sup sup S l J N ■ ■ ■ S l T ) N [V°}(x) *C sup S™[V°}(x) + Lr 



2| |2 



A 1 " ii.— .if 



for all t G [0, r ] and = x'PoX with P G /C. Finally we apply Lemma 4.2 to obtain 

the desired result. □ 



7. Further discussions and a numerical illustration 

7.1. Linear quadratic Hamiltonians. The contraction result being crucial to our anal- 
ysis (see Remark 5.4[ ), it is impossible to extend the results to the general case with linear 
terms as in |McE09j . However, the one step error analysis (Lemma 5.2) is not restricted 
to the pure quadratic Hamiltonian. Interested reader can verify that the one step error 
0(t 2 ) still holds in the case of [McE09j . Then by simply adding up the errors to time T, 
we get that: 

e(x, r, N, V°) < L(l + \x\ 2 )Nr 2 = L(l + \x\ 2 )Tt. 

Note that the term \x\ 2 is replaced by (1 + \x\ 2 ) for the general Hamiltonian with linear 
terms. This estimate is of the same order as in |McE09j with much weaker assumption, 
especially the assumption on E m . 



7.2. A tighter bound on the complexity. From Theorem 4.1 



tighter bound on the complexity of the algorithm (compared to [MK10J): 



and 15.11 we obtain a 



Corollary 7.1. Under Assumptions 2.1 and 3J}, to get an approximation of V of order 
e, the number of iterations is 



0(^i£), 



as e — > 0, 



whence the number of arithmetic operations is: 
(35) 0{\M\°(^n% 



as e — > 0. 
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7.3. Convergence time. Theorem 54^ shows that for a sufficiently small r and a prun- 
ing procedure of error r 2 , the discrete-time approximation error c :Pt (x, t, N, V°) is 0(t) 



uniformly for all N > 0. Meanwhile, by Theorem 4.1, the finite horizon approxima- 
tion error eq{x,T, V°) decreases exponentially with the time horizon T. Therefore, for a 
fixed sufficiently small r, the total error e (x, T, V°) + e Pr (x, r, N, V°) decreases at each 
propagation step and becomes stationary after a time horizon T such that 



(36) 



e (x,T,V°) ^e^(x,r,N,V°). 



If the estimate 0(t) is tight, then (36) implies that the stationary time T is bounded by 
the relation: 

0{e- aT ) < 0(t). 

Therefore the stationary time T is bounded by 0{— log(r)), which implies that nu- 
merically the total error stops decreasing after a number of iterations iV bounded by 
0(-log(r)/r). 

To give an illustration, we implemented this max-plus approximation method, incorpo- 



rating a pruning algorithm in |GMQ11] to a problem instance satisfying Assumption 2.1 
and 3.2 in dimension n = 2 and with \Ai\ = 3 switches. The pruning algorithm gener- 



ates an error of order at most r 2 at each step. We use the maximal absolute value of 



H(x, VV) (10) on the region [—2,2] x [—2,2] as the back-substitution error, denoted by 
|iy|oo> to measure the approximation. We observe that for each r, the back-substitution 
error \H\oo becomes stationary after a number of iterations, see Figure [l|for r = 0.0006. 
We run the instance for different r and for each r we collect the time horizon T when 

o 

-2 

_8 
g. -4 
_o 

-6 



-T=0.0006 




500 



1000 1500 2000 2500 3000 
N 



Figure 1. Plot of log |-ff|oo w.r.t. the iteration number N 



the back-substitution error becomes stationary. The plot shows a linear growth of T 
with respect to — log(r), which is an illustration of the exponential decreasing rate in 
Theorem 14.11 



1.5 



'1.5 5 5.5 6 6.5 7 7.5 
-log(T) 

Figure 2. Plot of the convergence time T w.r.t — log(r) 
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